Stanford researchers recently introduced tools to help users and developers understand large language models (LLM) in their totality. Given the central role of LLMs in NLP and in GenerativeAI, this suite of tools is an important step towards better transparency for language models. I hope other researchers build upon this exciting suite of techniques and ideas. When migrating to a modern data warehouse or data lakehouse, selecting the right table format is crucial. Brooklyn Data Company just released an important new benchmark comparing open source Delta Lake and Apache Iceberg.

Efficient Methods for Natural Language Processing: Roy Schwartz is Professor of NLP at The Hebrew University of Jerusalem. We discuss an important survey paper (co-written by Roy) that presents a broad overview of existing methods to improve NLP efficiency through the lens of NLP pipelines. Building a premier industrial AI research and product group .. in three years: Hung Bui is the CEO of Vietnam-based, VinAI, explains the process of building a team that within a span of three years found itself listed among the Top 20 Global Companies in AI Research. Stanford researchers recently introduced tools to help users and developers understand large language models (LLM) in their totality. Given the central role of LLMs in NLP and in GenerativeAI, this suite of tools is an important step towards better transparency for language models.

As we speak, we're thrilled to announce a local integration with Airbyte Cloud, which permits knowledge replication from any supply into Databricks for all knowledge, analytics, and ML workloads. Airbyte Cloud, a hosted service made by Airbyte, gives an integration platform that may scale along with your customized or high-volume wants, from giant databases to a long-tail of API sources. This integration with Databricks helps break down knowledge silos by letting customers replicate knowledge into the Databricks Lakehouse Vacation spot to course of, retailer, and expose knowledge all through your group. As an open supply normal for ELT, Airbyte gives greater than 150 editable pre-built connectors – or simply create new ones in a matter of hours. With a devoted Databricks connector, joint customers can sync any knowledge supply that Airbyte helps into Databricks Delta Lake.

There is also a new release of MLflow (MLflow 2.0), which is a machine learning operations platform for management of ML pipelines. In Databricks parlance, a Delta Lake represents a data architecture that has both storage and analytics capabilities; Data lakes store data in native format and a Data warehouse stores data in structured format (typically SQL). Hence, a delta lake is expected to be'one system – one copy' encapsulating both analytics and data in a single system.

At Insider, we use Apache Spark as the primary data processing engine to mine our clients' clickstream data and feed ML-ready data into our machine learning pipelines to enable personalizations. We have been using Spark since version 1.5 and always looking for ways to improve efficiency. If you are interested too, check out our blog post about how Spark 3 reduced our Amazon EMR cost by 40%. To further improve our platform's efficiency, we decided to conduct a trial with the Databricks platform. Before moving forward with the Databricks platform and the benchmarks, let's see how we utilize Apache Spark and Amazon EMR, and the pain points to understand better our current solutions and challenges.

For many organizations, real-time data collection and data processing at scale can provide immense advantages for business and operational insights. The need for real-time data introduces technical challenges that require skilled expert experience to build custom integration for a successful real-time implementation. For customers looking to implement streaming real-time applications, our partner Confluent recently announced a new Databricks Connector for Confluent Cloud. This new fully-managed connector is designed specifically for the data lakehouse and provides a powerful solution to build and scale real-time applications such as application monitoring, internet of things (IoT), fraud detection, personalization and gaming leaderboards. Organizations can now use an integrated capability that streams legacy and cloud data from Confluent Cloud directly into the Databricks Lakehouse for business intelligence (BI), data analytics and machine learning use cases on a single platform.

The latest winner of the growing interest in enterprise AI is Databricks, a startup that has just secured $1.6 billion in series H funding at an insane valuation of $38 billion. This latest round of investment comes only months after Databricks raised another $1 billion. Databricks is one of several companies that offer services and products for unifying, processing, and analyzing data stored in different sources and architectures. The category also includes Snowflake, which made a massive IPO last year and has a market cap of $90 billion, and C3.ai, another enterprise AI company that went public last year. Why are investors enamored with companies like Databricks?

Recently, I learned that the failure rate for machine learning projects is still astonishingly high. Studies suggest that between 85-96% of projects never make it to production. These numbers are even more remarkable given the growth of machine learning (ML) and data science in the past five years. For businesses to be successful with ML initiatives, they need a comprehensive understanding of the risks and how to address them. In this post, we attempt to shed light on how to achieve this by moving away from a model-centric view of ML systems towards a data-centric view. Of course, everyone knows that data is the most important component of ML. Nearly every data scientist has heard: "garbage in, garbage out" and "80% of a data scientist's time is spent cleaning data".

Databricks has launched a project to create an open-source data sharing protocol for securely sharing data across organisations in real time, independent of the platform on which the data resides. The Delta Sharing initiative, part of Databrick's open-source Delta Lake project, has already attracted support from a number of data providers, including NASDAQ, S&P and Factset, and leading IT vendors including Amazon Web Services, Microsoft and Google Cloud, according to Databricks. Databricks is also expanding its technology portfolio with a new machine learning system and the addition of new data pipeline and data governance capabilities to its flagship Databricks Lakehouse Platform, which combines aspects of data warehouse and data lake systems. Delta Sharing is the latest open-source initiative from Databricks, one of the most closely watched big data startups. Founded by the developers of the Apache Spark analytics engine, Databricks markets the Databricks Lakehouse Platform, its flagship unified data analytics platform.

In this presentation I first want to introduce Presto for those who don't know what it is and talk a little bit about Starburst and what we do here to help enterprise adoption of Presto. The main topic of my presentation is the Delta Lake integration that we've done for Presto and then sort of show how we can combine Presto and DataBreak, Spark, and Delta together in one data platform architecture and how to efficiently use the best, get the beauties of both technologies. And then finally show real use cases where that combination actually delivers the best results for your team. So with that let's get going. So Presto and Starburst, Presto itself is open source community driven project.